An Improved Merge-split Sampler for Conjugate Dirichlet Process Mixture Models

The Gibbs sampler is the standard Markov chain Monte Carlo sampler for drawing samples from the posterior distribution of conjugate Dirichlet process mixture models. Researchers have noticed the Gibbs sampler’s tendency to get stuck in local modes and, thus, poorly explore the posterior distribution. Jain and Neal (2004) proposed a merge-split sampler in which a naive random split is sweetened by a series of restricted Gibbs scans, where the number of Gibbs scans is a tuning parameter that must be supplied by the user. In this work, I propose an alternative merge-split sampler borrowing ideas from sequential importance sampling. My sampler proposes splits by sequentially allocating observations to one of two split components using allocation probabilities that are conditional on previously allocated data. The algorithm does not require further sweetening and is, hence, computationally efficient. In addition, no tuning parameter needs to be chosen. While the conditional allocation of observations is similar to sequential importance sampling, the output from the sampler has the correct stationary distribution due to the use of the Metropolis-Hastings ratio. The computational efficiency of my sequentially-allocated merge-split (SAMS) sampler is compared to Jain and Neal’s sampler using various values for the tuning parameter. Comparisons are made in terms of autocorrelation times for four univariate summaries of the Markov chains taken at fixed time intervals. In four examples involving different models and datasets, I show that my merge-split sampler usually performs substantially better — in some cases, two to five times faster — than existing methods, and never performs worse.